Method and apparatus for cluster interconnection using multi-port nodes and multiple routing fabrics

ABSTRACT

Interconnect networks are described that allow nodes having more than two ports to be interconnected. More particularly, each node interfaces with multiple and more than two, fabrics. Also, all fabrics are incomplete in that not every node interfaces with every fabric, and no fabric includes all the nodes, yet every pair of nodes appears together in at least one fabric. Nodes are used that appear together in a fabric as a class of nodes that exhibit similar interconnection properties. The present invention allows for scalable, high-performance and reliable interconnection of large numbers of end nodes while satisfying constraints on architecture of end nodes and networking equipment. Bounds for the number of fabrics and fabric size are disclosed for designing an optimized interconnection network.

REFERENCE TO EARLIER APPLICATION

[0001] This application claims the benefit of U.S. ProvisionalApplication No. 60/393,936 filed Jul. 2, 2002 which is hereinincorporated by reference.

FIELD OF THE INVENTION

[0002] The present invention relates to the field of digitalcommunication. More particularly, the present invention relates to amethod and apparatus implementing a communications network.

BACKGROUND OF THE INVENTION

[0003] In modem computer systems, much of their functionality isrealized by the ability to network, that is connect, various computersto provide digital communication. Indeed many interconnection schemeshave been developed that meet certain needs in various ways. Forexample, multiprocessor systems can be configured as bus-connected orring-connected multiprocessor systems. However, the operation and designconstraints of such systems do not lead to designs for reliable andscalable switched networks, especially ones that implement crossbarswitches employing wormhole routing. The primary limitation of this typeof configuration is that ring topologies are not suitable forwormhole-routed switched networks and result in an unacceptably largehop count between end nodes as the number of nodes is scaled.

[0004] In another example, the design of bus-oriented interconnectiontopologies for single-hop communication among multiple transceiverstations is not applicable to scalable switched networks because, amongother things, a single-hop interconnection between a large number ofnodes is impossible when crossbar switches with a limited number ofports are used. Moreover, such designs use bus-based interconnects whichbear little resemblance, if any, to switched interconnects.

[0005] Non-bus-oriented single-hop interconnections are also deficientin a number of ways. For example, such configurations suffer the samelimitations as described above while also connecting nodes (orswitchless networks) directly. This latter feature limits theapplicability of the design to end nodes having a large number of portsand to fabrics having zero switches and hence is inapplicable to thedesign of switched interconnects.

[0006] In a traditional approach, ServerNet networks have been designedwith two ports, also called colored ports or “X” and “Y” ports,connected to two complete, independent groups of crossbar switches. Theinterconnection group is complete because every end node interfaces witheach group of crossbar switches and each group of switches interfaceswith every node. Moreover, the interconnection group is independentbecause ports of one type are only connected to other ports of the sametype. For example, each of the X ports is only connected via an X fabricto other X ports and each of the Y ports in the network are likewiseonly connected via a Y fabric to other Y ports. Note here that an Xfabric is a group of switches that connect all the X ports and only theX ports in the network (similarly for Y ports). In this way, a fabric ofone type is designed independently of other fabrics of other types.

[0007] A particular concern in network design is fault tolerance. With alarge scaled system there is insufficient protection against singlepoints of failure because of the large number of components and it ishard to maintain symmetry because of failed parts. Moreover, scalabletopologies (e.g. fat trees) offer design points exponentially far apart.In addition, the relative capacity of an end node shrinks as a networkgrows in size.

[0008] One improved approach has introduced ServerNet AsymmetricFabrics. With this approach, end nodes are connected using two completebut non-identical groups of switches. However, network expansionrequires scalable switched networks. Switched processor-memorysubsystems include Sun UEIOK, Intel Profusion Chipset, Compaq Alpha EV7.Switched I/O subsystems include ServerNet, NGIO (Next Generation I/O),and Future I/O. However, the issue is scalable yet highly availablefabrics. Hence, there is a further need for optimizing the reliabilityand performance of scalable switched networks.

[0009] There exists prior art in the area of bus-connected andring-connected multi-computer systems, however, the operation and designconstraints of such prior art does not lead to designs for reliable andscalable switched networks, especially networks configured for use withcrossbar switches employing wormhole routing. Moreover, the prior artdoes not address how a network comprising multiple incomplete fabricscan simultaneously optimize the reliability and the performance ofscalable switched networks.

[0010] While the above interconnection schemes provide certainfunctionality, they are nonetheless limited in at least the waysdiscussed above. With the advent of network interface cards and othersimilar devices that provide for multiple ports on one computer system,network design can be expanded beyond the constraints of prior artsystems. Importantly, interconnection fabrics need not be constrained tobeing complete nor colored. Notably, interconnection fabrics can beallowed to be incomplete while allowing for improved fault tolerancewhile using reduced hardware resources. Toward finding an optimaldesign, however, there exists a need to determine bounds on variousparameters of network designs.

SUMMARY OF THE INVENTION

[0011] In an embodiment of the invention, an interconnect network isdesigned so as to allow nodes having more than two ports to beinterconnected. More particularly, in an embodiment of the invention,each node interfaces with more than two fabrics. Also, all fabrics areincomplete in that not every node interfaces with every fabric, and nofabric interfaces with all the nodes, yet every pair of nodes interfacesto at least one common fabric. The present invention uses nodes thatappear together in a fabric as a class of nodes that exhibit similarinterconnection properties. The present invention allows for scalable,high-performance and reliable interconnection of large numbers of endnodes while satisfying constraints on architecture of end nodes andnetworking equipment including fabrics that comprise n-port crossbarswitches.

[0012] In one embodiment of the invention, a multi-fabricinterconnection system is disclosed, comprising k interconnectionfabrics and n nodes. The k interconnection fabrics each have a vertexcardinality, m. The n nodes each have p ports, wherein p is greater thantwo, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics.Also, each of the k interconnection fabrics interconnects less than allof the n nodes. In this embodiment of the invention, the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$

[0013] In another embodiment, the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$

[0014] In yet another embodiment, the optimal fabric size, m_(O), isbounded above by the relationship$m_{o} \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$

[0015] And in still another embodiment, the optimal fabric size, m_(O),is bounded below by the relationship$m_{o} \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$

[0016] In another embodiment, each fabric provides redundantconnectivity to all nodes to allow continued connectivity in spite ofthe failure of part of a network.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and form apart of this specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

[0018]FIG. 1 is network diagram for interconnecting seven nodes eachwith three ports according to an embodiment of the present invention.

[0019]FIG. 2 is a network diagram for interconnecting three elementsusing three fabrics.

[0020]FIG. 3 is a network diagram for interconnecting four elementsusing six fabrics.

[0021]FIG. 4 is a network diagram for interconnecting five elementsusing ten fabrics.

[0022]FIG. 5 is a network diagram for connecting 65 nodes using fiveelements and ten fabrics.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In order to understand the present invention, it is useful tounderstand combinatorial block design and, in particular, balancedincomplete block design (BIBD). A block is a subset, s, of a set ofelements, S, where block design considers choosing blocks with certainproperties. A block design is called incomplete if at least one blockdoes not contain all of the set elements. A block design is balanced ifeach block has the same number of elements and each pair of elementsoccurs in a block the same number of times. In the present invention,BIBDs are used to design networks that have predeterminedcharacteristics or properties.

[0024] A BIBD is a pair (V, B) where V is a set of v elements and B is acollection of b blocks that are subsets of k elements of V such thateach element of V is contained in exactly r blocks and any two-subsetsof V is contained in exactly λ blocks. The variables v, b, r, k, and λare parameters of a BIBD also referred to as 2−(v, b, r, k, λ) blockdesigns. Two necessary conditions must be met for the existence of aBIBD: (1) vr=bk, and (2) r(k−1)=λ(v−1). A consequence of theseconditions is that three parameters, v, k, and λ, determine theremaining two parameters, r and b, where

r(k−1)=λ(v−1) $\begin{matrix}{{{r = \frac{\lambda \left( {v - 1} \right)}{k - 1}},{and}}{{bk} = {vr}}} & \text{(Equation~~1)} \\{b = {\frac{vr}{k}.}} & \text{(Equation~~2)}\end{matrix}$

[0025] With regard to Equation 1, consider that an element, x, occurs inr blocks. Further consider that each of the blocks, x, is paired withk−1 elements. Thus, we have r(k−1) pairs. Note that x is paired with allother v−1 nodes exactly A times (i.e., A (v−1)) and Equation 1 istherefore proven. It is straightforward to see that each block, b,contains k elements for a total of bk elements. Also, each elementoccurs in r blocks and since there are v elements the total is vr, thuswe have Equation 2.

[0026] Accordingly, a BIBD (v, b, r, k, λ) design can also be referredto as a (v, k, λ) design. The notation 2−(v, k, λ) is also used, sinceBIBDs are t-designs of the form t−(v, k, λ) with t=2. Note that when λ1(i.e., 2−(v, k, 1)), the notation S(2, k, v) is also used denoting thatthese are Steiner systems (named after nineteenth century geometer JakobSteiner). With regard to Steiner systems, given three integers, t, k, v,such that 2≦t<k<v, a Steiner system S(t, k, v) is a set V of v elementstogether with a family, B, of subsets of k elements of V (i.e., blocks)with the property that every subset of t elements of S is contained inexactly one block. Recall that in BIBD, t=2. These systems thereforedetermine the number of groups that are needed to connect v items,arranged in groups of k, such that a pair (i.e., “2−”) appears inexactly λ groups, where in a Steiner system λ=1 group.

[0027] Moreover, from Fisher's inequality, b≧v, designs with b=v and r=kare called symmetric designs where every block contains k elements andevery element occurs in r blocks. Also, every pair of elements occurs inλ blocks, and every pair of blocks intersects in λ elements.

[0028] Whereas BIBD designs can be quite complicated they can berepresented in a two-dimensional, k×b array in which each columncontains the elements forming a block. For example, consider the 2−(9,3, 1)=12 design:${Elements}\left\{ \overset{\overset{\text{Block}}{}}{\begin{matrix}0 & 0 & 0 & 0 & 1 & 1 & 1 & 2 & 2 & 2 & 2 & 6 \\1 & 3 & 4 & 5 & 3 & 4 & 5 & 3 & 4 & 5 & 4 & 7 \\2 & 6 & 8 & 7 & 8 & 7 & 6 & 7 & 6 & 8 & 5 & 8\end{matrix}.} \right.$

[0029] Here, for example, the first column represents the blockcontaining elements e₀, e₁, and e₂ and the twelfth column represents ablock having elements e₆, e₇, and e₈. In a larger design, letters can beused to represent blocks with more than 10 elements. The sequence 0, 1,. . . , 9, a, b, . . . , z can represent designs with up to 36 elements(i.e., 10 numerically represented elements and 26 alphabeticallyrepresented elements). Thus, the following 2−(16, 4, 1)=20 design can berepresented as follows:$\quad{{Elements}{\overset{\overset{Block}{}}{\left\{ \begin{matrix}0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 1 & 2 & 2 & 2 & 2 & 3 & 3 & 3 & 3 & 4 & 5 & 6 \\1 & 4 & 7 & a & d & 4 & 5 & 6 & 9 & 4 & 5 & 6 & 8 & 4 & 5 & 6 & 7 & 8 & 9 & 7 \\2 & 5 & 8 & b & e & 7 & b & 8 & c & c & 7 & 9 & a & 9 & 8 & a & b & b & a & c \\3 & 6 & 9 & c & f & a & d & e & f & e & f & b & d & d & c & f & e & f & e & d\end{matrix} \right.}.}}$

[0030] With a design in hand, a BIBD can be further described by anincidence matrix A which has the blocks as its columns and elements(i.e., nodes) as the rows. Thus, an entry, a_(i,j) of the incidencematrix A is equal to one if the ith element resides in the jth block,otherwise it is equal to zero. For example, for a symmetric design withN elements, the incidence matrix is an N×A matrix. Accordingly, the2−(9, 3, 1)=12 design${Elements}\left\{ \overset{\overset{\text{Block}}{}}{\begin{matrix}0 & 0 & 0 & 0 & 1 & 1 & 1 & 2 & 2 & 2 & 3 & 6 \\1 & 3 & 4 & 5 & 3 & 4 & 5 & 3 & 4 & 5 & 4 & 7 \\2 & 6 & 8 & 7 & 8 & 7 & 6 & 7 & 6 & 8 & 5 & 8\end{matrix}} \right.$

[0031] described above is represented by the following incidence matrix$A_{2 - {({9,3,1})}} = {\overset{\overset{Block}{}}{\left. \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\1 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\1 & 0 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 \\0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 1 & 0 \\0 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 \\0 & 0 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & 0 & 1 \\0 & 1 & 1 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 1\end{bmatrix} \right\}}{Incidence}\quad {of}\quad {Elements}}$

[0032] From a BIBD, network designs can in turn be generated byidentifying certain correspondences. For example, given the blocks of aBIBD (v, k, λ), the mapping between BIBD and network design are given bythe following table. TABLE 1 Mapping from Block Design to Network DesignBlock Design Network Design Elements Nodes Blocks Fabrics λ (i.e.,pairwise occurrence) Fabrics between each pair r (i.e., total occurrenceof an element) Degree of a node k (i.e., block size) Length of routing

[0033] A solution to a BIBD provides a partition of the v nodes intosubsets such that there are exactly λ subsets for each pair of the nodesand the distance between any two nodes is at most k−1. The firstproperty ensures that the target topology for the network is stronglyconnected. The second property is important to obtain a bound on themaximum routing length which is measured by the number of hops. Eachblock of a BIBD leads to one or two virtual rings (depending on theembedding technique), and each node of the network has one or morevirtual nodes on each virtual ring. A virtual node is created when avirtual ring traverses a physical node.

[0034] Thus, the two important parameters of a block design are k and r.The size k of each block determines the maximum length of routing andthe total number of occurrences, r, of each node determines the degreerequirement for such node in the target network. Particularly, smaller kleads to a better bound on the length of routing and smaller r requiresa smaller number of ports and links on the target network.

[0035] For λ=1 (i.e., a Steiner system), each block of size k is uniquefor all possible pairs of k nodes that it contains. That implies thateach possible pairing of nodes in a block corresponds to a uniquecandidate edge for the target topology. Furthermore, since such an edgenever occurs in any other block, the virtual rings are mutuallyedge-disjoint. Thus, each block of size k can induce a complete graph ofk nodes. In graph theory, any graph with k nodes can be embedded into acomplete graph with k nodes.

[0036] Using the foregoing principles, the present invention provides anew class of interconnect networks and multiple incomplete fabricinterconnect systems that can be used to simultaneously scale theperformance and the reliability of either multi-computer cluster systemsor switched input/output systems, while using fewer components than atraditional approach. In doing so, each end node, such as a computer ornetwork-attached I/O device, has more than two network interface ports.The multiple ports can be provided either through the use of computerswith network interface cards (NICs), each having one or more ports, orthrough the use of multi-port I/O nodes. Preferably, this approach takesadvantage of the dual-ported NICs that that are a key part of widelyused networks including, for example, ServerNet networks designed by theHewlett-Packard Corporation. Such an approach can also be implemented innetworks including Ethernet, GigaNet, Fiber Channel, ATM (AsynchronousTransfer Mode), RDMA-enabled Ethernet, PCI Xpress, InfiniBand or othernetworks to be developed in the future.

[0037] To better understand the present invention, consider that afabric is a collection of routers and links that interconnect a set ofnodes. A node may have one or more NICs (network interface cards), eachwith two or more ports. Among other things, each port allows a node tobe on a distinct fabric. In one embodiment of the invention, fabrics,ports and routers have color restrictions. For example, ports androuters are either red or green (note that the coloring described herecan also be described with reference to X and Y designations). In acoloring scenario case, it is illegal to connect a red port or router toa green port or router; each fabric connects either red ports using onlyred routers (i.e., a red fabric) or green ports using only green routers(i.e., a green fabric), but there is no interconnection between colors.The larger problems underlying network topology design are wellunderstood—minimizing diameter, maximizing bisection width, minimizingthe number of routers, avoiding excessive link contention and avoidinghot links—and assumed to be important here. In other embodiments of theinvention, coloring constraints are eliminated.

[0038] Several non-trivial issues unique to multi-fabric topologies willnow be examined. More particularly, a determination of how large eachfabric needs to be will be examined. As a fundamental matter, fabricsmust collectively provide at least one path between each pair of nodes.While this can be accomplished with a large number of fabrics, a numberof fabrics larger than necessary can waste routers, thereby increasingcosts, by making redundant connections between nodes.

[0039] A determination of how many fabrics are needed is also important:This is an important yet difficult matter to determine. In oneembodiment of the invention, this number of fabrics is bounded aboveand/or below to determine an approximation for an optimal solution. Asbefore, this will ensure that each pair of nodes appear together in atleast one fabric, given a specific fabric size.

[0040] It is evident that redundant connections are inevitable in allbut the simplest of cases. Should redundant connections be present, apair of nodes will co-occur in more than one fabric. Further, withineach fabric, distance between nodes will vary from pair to pair. Ratherthan have a pair of nodes be far apart in all fabrics—and have otherpairs be close together in more than one fabric—nodes within a fabricshould be so arranged that distances between otherwise far nodes becomesmaller, perhaps at the expense of increasing the distance betweenotherwise closely connected nodes. Given f, the number of nodes that fiteasily within a COTS (commercial off-the-shelf) rack frame, it isdesirable that the number of wires that need to travel between frames bekept within reasonable limits.

[0041] It should be noted that the multi-fabric design problem discussedhere is different from the problem of multiple ports in one fabric. Forexample, multiple fabrics according to the present invention are likelyto provide better protection for nodes against faults and congestion.Moreover, the diameter of a multi-fabric network according to thepresent invention is generally smaller than that of its single-fabriccounterpart. This not only reduces the number of outstanding packetsnecessary for keeping pipelines full but also lessens the impact ofoutput-port contention on link utilization. In effect, the multiplefabrics of the present invention create congestion-containment domains.

[0042] To further understand the present invention, consider thefollowing problem: given n nodes, each node connecting top differentfabrics, what is (1) the minimum number of fabrics and (2) the minimumfabric cardinality required to ensure full connectivity between allnodes? Furthermore, what is a minimal assignment of connections tofabrics?

[0043] While the present invention is appropriate for colored andnon-colored fabric implementation, implementations that completelyignore color will be first considered. In doing so, it has been foundthat n nodes can be connected using k fabrics of vertex cardinality msuch that $\begin{matrix}{\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil \leq k} & \left( {{Equation}\quad 3} \right)\end{matrix}$

[0044] where ┌•┐ represents rounding up to the next whole number and$\quad\begin{pmatrix}i \\j\end{pmatrix}$

[0045] represents the binomial coefficient C^(i) _(j) such that$C_{j}^{i} = {\frac{i!}{{j!}{\left( {i - j} \right)!}}\overset{\Delta}{=}\begin{pmatrix}i \\j\end{pmatrix}}$

[0046] denotes the number of sub-populations of size j that can bechosen from a set of size i (i.e., i choose j). The above inequalityfollows from the requirement that every pair of nodes must be connectedby at least one fabric. Moreover, since each fabric generates at most$\quad\begin{pmatrix}m \\2\end{pmatrix}$

[0047] pairs, and connectedness requires at least $\quad\begin{pmatrix}n \\2\end{pmatrix}$

[0048] pairs, the resulting lower bound on the number of fabrics, k,follows.

[0049] In now considering the lower bound on fabric size, theneighborhood relationships of a single node are examined to impose aconstraint that a node has to connect to all of its peers througha-finite (and preferably small) number of ports. Using concepts fromgraph theory, consider that a node forms a vertex on a graph and an edgeis an unordered pair of distinct vertices. It has therefore been foundthat with n nodes, each with p ports connected using fabrics of vertexcardinality m (i.e., the number of vertices), then $\begin{matrix}{m \geq {\left\lceil \left( \frac{n + p - 1}{p} \right) \right\rceil.}} & \left( {{Equation}\quad 4} \right)\end{matrix}$

[0050] First, because a node has only p ports, it cannot connect to morethan p fabrics. Moreover, because each fabric offers connections to onlym−1 other neighbors, m must be large enough to cover all neighbors.Therefore, $\left\lceil \frac{n - 1}{m - 1} \right\rceil \leq p$

[0051] such that

(n−1)≦p(m−1)

(n−1)≦pm−p

[0052] which we manipulate into the form$m \geq {\left\lceil \left( \frac{n + p - 1}{p} \right) \right\rceil.}$

[0053] A straightforward example reinforces the above aspects of thepresent invention. As shown in FIG. 1, consider interconnecting 7 nodeseach with 3 ports. In fact, this problem corresponds to the BIBD of2−(7, 3, 1)=7, that is, seven (7) groups are needed to connect seven (7)items, arranged in groups of three (3), such that a pair appears inexactly one (1) group. Since each node must communicate to its 6 peersvia only 3 ports, each fabric must have a size (i.e., vertexcardinality) at least 3, according to Equation 4:$m \geq \left\lceil \frac{7 + 3 - 1}{3} \right\rceil$ m ≥ 3.

[0054] Moreover, the minimum number of fabrics, according to Equation 3is $\left\lceil \frac{\begin{pmatrix}7 \\2\end{pmatrix}}{\begin{pmatrix}3 \\2\end{pmatrix}} \right\rceil \leq k$ 7 ≤ k.

[0055] In this example, it is important to note that these lower boundsprovide tight bounds. Indeed, the fact that both these lower bounds aretight, at least for certain cases, is illustrated by an assignment ofnodes to fabrics as shown in Table 2. TABLE 2 Assignment of Nodes FabricNode 1 Node 2 Node 3 1 1 2 3 2 1 4 5 3 1 6 7 4 2 4 6 5 2 5 7 6 2 5 6 7 34 7

[0056] This shows that 7 fabrics of size 3 are not merely the minimumrequirement but are also sufficient in this case. This interconnectionof fabrics is further shown in FIG. 1.

[0057] It is found that the coloring of fabrics adds strong constraintsto the fabric partitioning problem. In fact, multi-fabric design withnodes having only two ports, each a different color, is impractical inall but the most trivial cases. Consider that if each node has twoports, one red and one green, then at least one fabric must connect allthe nodes. This result can be shown by contradiction. For example,suppose that a node n connects to a red fabric F_(R) and a green fabricF_(G) in such a fashion that neither F_(R) nor F_(G) connects all thenodes together, that is,

[0058] F_(R)⊂N and

[0059] F_(G)⊂N

[0060] where N is the set of all nodes. Thus, either

[0061] F_(R)∪F=N

[0062] or

[0063] F_(R)∪F_(G) is a strictly proper subset of N.

[0064] Since the latter would imply incomplete connectivity for n, onlythe former can be accepted. Therefore,

[0065] F_(R)∪F_(G)=N.

[0066] Since node n belongs to both red and green fabrics, there mustexist nodes

[0067] n_(R)εF_(R)

[0068] and

[0069] n_(G)εF_(G)

[0070] such that

[0071] n_(R)≠n_(G),

[0072] n_(R∉n) _(G), and

[0073] n_(G)∉F_(R).

[0074] In order to achieve complete connectivity between all pairs ofnodes, it is therefore necessary to add a fabric, say F_(X), that willconnect n_(R) to n_(G) where F_(X) could be neither red nor green.Because, it is impossible to connect n_(R) to n_(G) using coloredfabrics as constrained above, a contradiction exists. The only availableports for connecting to F_(X) on n_(R) is green and on n_(G), red.Because our supposition has been contradicted, the opposite must betrue, that is, at least one fabric must connect all the nodes.

[0075] It is because of this result that multi-fabric design was notattempted in prior art systems with only two ports, such as ServerNet I.With the availability of multi-port equipment, such as dual-PCI CompaqProfessional Workstation platforms that support two NICs each with twoports, called the X and Y ports in ServerNet parlance, multi-fabricdesigns are now feasible and, indeed, desirable for their advantages.With regard to a particular embodiment of the invention, it has beenfound that ServerNet II offers a flexible coloring of ports so that evenwith only one ServerNet II NIC, a node can have two ports of the samecolor. Partitioned fabric designs according to the present invention aretherefore practical even in systems having only one ServerNet II NIC pernode, but not practical in systems with only one ServerNet I NIC pernode.

[0076] The further advantage of ServerNet II's flexible coloring of NICports becomes apparent when the fabric-partitioning solution describedin Table 2 is examined. If all ports were the same color, the solutiondescribed above would function properly because fabric coloring wouldnot be an issue. For nodes with a pair of ServerNet I NICs, however, twoof the four ports on each node are X ports and two are Y ports.ServerNet I NICs and routers set and check the path bit in almost allpackets (except for default ports on routers) and, in general, it is notpossible to route packets between X and Y ports and/or routers. Withregard to Table 2, rows of the table (or, fabrics) need to be colored insuch a way that no node appears in more than two fabrics of the samecolor.

[0077] Let us now consider a specific impossibility argument in thecontext of Table 2 and then a general theorem for partitions with an oddnumber of fabrics. Without loss of generality, suppose that a fabric,say Fabric 1, is colored red. Since Node 1 has only two red ports and itappears on a total of three fabrics, it must be that at least one of theother two fabrics on which it appears must be green. Again, without lossof generality, suppose that a second fabric, say Fabric 2, is paintedgreen. Applying the same argument to Node 2, either Fabric 4 or Fabric 5must be green. Suppose that Fabric 4 is green. Next, consider Fabric 7.Since both the green ports on Node 4 are used up, this fabric must bepainted red. Doing so uses up both the red ports on Node 3. Hence,Fabric 6 must be painted green. Doing so uses up both the green ports atNode 5. Hence, Fabric 5 must be painted red. Now, we need to assign acolor to Fabric 3 which connects Nodes 1, 6 and 7, but both green portsare used up on Node 6 as well as both red ports on Node 7. It istherefore impossible to pick a color for Fabric 3.

[0078] In proceeding, we will further be constrained by the fact that itis not possible to color an odd number of fabrics with two colors—say,red and green—if each node has an equal number of red and green portsand the node-to-fabric assignment.

[0079] Having now considered lower bounds, to be able to grasp upperbounds is also desirable. Although redundancy may be inevitable,redundancy can be quantified by fixing at the outset the number of nodesthat will co-occur in all fabrics. Optimal solutions will not always bepossible, but an interesting effect is that we can always come up with afeasible solution. Since the solutions so found yield closed-formexpressions for the size and the number of fabrics, those expressionsserve as upper bounds on the respective quantities. The key observationhere is that many nodes may connect to the same collection of fabrics,and these equivalent nodes can be handled together in an equivalenceclass. Equivalence classes can be thought of as nodes that alwaysco-occur in fabrics. Equivalence classes are a natural algebraicabstraction for the multi-fabric design problem because connectedness,the primary relationship of interest here, is, algebraically speaking,an equivalence relation in that it is trivially reflexive, symmetric andtransitive. A solution is constructed by increasing the number ofequivalence classes. For the degenerate and trivial cases of one or twoclasses, a single fabric connects all nodes, and each node needs onlyone fabric connection. That stated, we turn to more useful designs.

[0080] Partitioning nodes into three equivalence classes, S₁, S₂ and S₃,as shown in FIG. 2, each node has two connections and there are threetotal fabrics. Fabric F₁₂ connects all the nodes in S₁ and S₂, FabricF₁₃ connects all nodes in S₁ and S₃, and Fabric F₂₃ connects all nodesin S₂ and S₃. With four equivalence classes, as shown in FIG. 3, eachnode connects to three fabrics and there are $\begin{pmatrix}4 \\2\end{pmatrix} = 6$

[0081] different fabrics in all. With five equivalence classes, as shownin FIG. 4, each node connects to four fabrics and there are$\begin{pmatrix}5 \\2\end{pmatrix} = 10$

[0082] total fabrics.

[0083] More particularly, the network of FIG. 4 can be considered for aninterconnection of a 64-node cluster with four connections. In anembodiment this is achieved with two SHIP cards, each with an X port anda Y port. With these specifications, the network of FIG. 5 is built. Inorder to simplify design, the nodes are partitioned into equivalenceclasses where each fabric is a pairing of equivalence classes. With fiveequivalence classes, S1-S5 as shown in FIG. 5, each node connects tofour fabrics and there are ten total fabrics. Rounding the number ofnodes up to 65, we have 13 (i.e., 65/5=13) nodes per class with eachfabric connecting 26 (i.e., 2×13=26) nodes. Note that if each fabricwere a simple Steiner tree, 26 nodes would require 6 routers such thatthe 64-node configuration can be done in 6*10=60 routers. The completesolution is therefore shown in FIG. 5. Coloring constraints are easilysatisfied because the perimeter of the pentagon can be built with Xfabrics (shown as solid lines) and the core with Y fabrics (shown asdashed lines). Indeed, an important result of the present invention isthat it provides for fault-tolerant systems; the occurrence of a failureanywhere in the system will not render the rest of the system useless.Moreover, the present invention provides for redundant interconnectionpaths such that if a fail does occur, a redundant path is available.

[0084] Indeed, this solution can be extended to more general networkconfigurations with the understanding of equivalence classes. For nodeswith p ports, there are (p+1) equivalence classes. With n such nodes,the vertex cardinality of each fabric is given by$P = {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$

[0085] This solution is better understood with a full understanding ofequivalence classes. The resulting network topologies are interesting.Bisection bandwidth is observed to be good for these topologies, but itcan be difficult to compute because the number of classes is usuallyodd. Using a tree for each fabric, the 64-node topology discussed abovehas a bisection bandwidth of greater than ten (10) links. Because of thehigh cost of the 60 routers, adoption of such a design can be difficult.The quality of solutions generated—as quantified by, say, bisectionwidth and number of routers needed—depends upon the size of equivalenceclasses. The smaller the class size, the lesser is the number ofconnections that repeat in all fabrics. Because each node participatesin p fabrics, the connections within a node's equivalence class areredundantly repeated (p−1) times. Thus, it can be seen that the largerthe class size, the greater is the waste. Since each fabric connectsonly two classes (which is an arbitrary restriction), given the lowerbounds on fabric size discussed above, class size must be at least${{Class}\quad {size}} \leq {\left\lceil \frac{n + p - 1}{2p} \right\rceil.}$

[0086] For the 64-node topology, the bounds of the present inventionsuggest a minimum fabric size of 17, and a minimum fabric count of 15.At the fabric size of 26, a large enough fabric size is used, but,subjectively, a large number of fabrics may be in use. At this fabricsize, the lower bound on the number of fabrics is 7. At 10 fabrics, thenetwork is significantly above the minimum necessary. Whereas thepresent invention confirms the existence of a feasible solution, itsresults can be less than optimal. Moreover, the present invention can beused to contrast its choice of fabric parameters against lower boundsand also to state upper bound results. Furthermore, it is guaranteedthat the hard-to-satisfy color constraint of multi-fabric partitioningwill be satisfied in designs according to the present invention.

[0087] The bounding results of the present invention are thereforesummarized here for the optimal fabric size Fabric Parameter Lower BoundUpper Bound Optimal Fabric Size (m_(O))$\left\lceil \frac{n + p - 1}{p} \right\rceil$

$2\left\lceil \frac{n}{p + 1} \right\rceil$

[0088] and the optimal number of fabrics Fabric Parameter Lower BoundUpper Bound Optimal Number of Fabrics (k_(O))$\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil\quad$

$\begin{pmatrix}{p + 1} \\2\end{pmatrix}\quad$

[0089] In an embodiment, the present invention is practiced on acomputer system 600 as shown in FIG. 6. Referring to FIG. 6, anexemplary computer system 600 (e.g., personal computer, workstation,mainframe, etc.) upon which the present invention may be practiced isshown. When configured to practice the present invention, system 600becomes a computer aided design (CAD) tool suitable for assisting indesigning interconnect systems in large and small scale applications.Computer system 600 is configured with a data bus 614 thatcommunicatively couples various components. As shown in FIG. 6,processor 602 is coupled to bus 614 for processing information andinstructions. A computer readable volatile memory such as RAM 604 isalso coupled to bus 614 for storing information and instructions for theprocessor 602. Moreover, computer readable read only memory (ROM) 606 isalso coupled to bus 614 for storing static information and instructionsfor processor 602. A data storage device 606 such as a magnetic oroptical disk media is also coupled to bus 614. Data storage device 606is used for storing large amounts of information and instructions. Analphanumeric input device 610 including alphanumeric and function keysis coupled to bus 614 for communicating information and commandselections to the processor 602. A cursor control device 612 such as amouse is coupled to bus 614 for communicating user input information andcommand selections to the central processor 602. Input/outputcommunications port 616 is coupled to bus 614 for communicating with anetwork, other computers, or other processors, for example. Display 618is coupled to bus 614 for displaying information to a computer user.Display device 618 may be a liquid crystal device, cathode ray tube, orother display device suitable for creating graphic images andalphanumeric characters recognizable by the user. The alphanumeric input610 and cursor control device 612 allow the computer user to dynamicallysignal the two dimensional movement of a visible symbol (pointer) ondisplay 618.

[0090] While various embodiments and advantages of the present inventionhave been described, it will be recognized that a number of variationswill be readily apparent. For example, in implementing equivalenceclasses the present invention can be scaled to implement networks ofmany sizes. Thus, the present teachings may be widely applied consistentwith the foregoing disclosure and the claims which follow

What is claimed is:
 1. A multi-fabric interconnection system,comprising: k interconnection fabrics each having a vertex cardinality,m; and n nodes each having p ports, wherein p is greater than two, andwherein every pair of nodes from among the n nodes is interconnectedthrough at least one of the k interconnection fabrics, wherein each ofthe k interconnection fabrics interconnects less than all of the n nodes2. The interconnection system of claim 1, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


3. The interconnection system of claim 1, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


4. The interconnection system of claim 1, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


5. The interconnection system of claim 1, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {2{\left\lceil \frac{n + p - 1}{p} \right\rceil.}}$


6. The interconnection system of claim 1, wherein the k interconnectionfabrics provide redundant interconnection paths between every pair ofnodes from among the n nodes.
 7. A multi-fabric interconnection system,comprising: k interconnection fabrics each having a vertex cardinality,m; and n nodes each having p ports, wherein p is greater than two, andwherein every pair of nodes from among the n nodes is interconnectedthrough at least one of the k interconnection fabrics, wherein each ofthe k interconnection fabrics interconnects less than all of the nnodes, and wherein the number k of interconnection fabrics is boundedabove by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


8. The interconnection system of claim 7, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


9. The interconnection system of claim 7, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


10. The interconnection system of claim 7, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


11. The interconnection system of claim 7, wherein the k interconnectionfabrics provide redundant interconnection paths between every pair ofnodes from among the n nodes.
 12. A multi-fabric interconnection system,comprising: k interconnection fabrics, each having a vertex cardinality,m; and n nodes each having p ports, wherein p is greater than two, andwherein every pair of nodes from among the n nodes is interconnectedthrough at least one of the k interconnection fabrics, wherein everypair from among the p ports is interconnected to at least one of kinterconnection fabrics, wherein each of the k interconnection fabricsinterconnects less than all of the n nodes, and wherein the number k ofinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


13. The interconnection system of claim 12, wherein the number k of theinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


14. The interconnection system of claim 12, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


15. The interconnection system of claim 12, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


16. The interconnection system of claim 12, wherein the kinterconnection fabrics provide redundant interconnection paths betweenevery pair of nodes from among the n nodes.
 17. A multi-fabricinterconnection system, comprising: k interconnection fabrics eachhaving a vertex cardinality, m; and n nodes each having p ports, whereinp is greater than two, and wherein every pair of nodes from among the nnodes is interconnected through at least one of the k interconnectionfabrics.
 18. The interconnection system of claim 17, wherein the numberk of interconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


19. The interconnection system of claim 17, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


20. The interconnection system of claim 17, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


21. The interconnection system of claim 17, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


22. The interconnection system of claim 17, wherein the kinterconnection fabrics provide redundant interconnection paths betweenevery pair of nodes from among the n nodes.
 23. A multi-fabricinterconnection system, comprising: k interconnection fabrics eachhaving a vertex cardinality, m; and n nodes each having p ports, whereinp is greater than two, and wherein every pair of nodes from among the nnodes is interconnected through at least one of the k interconnectionfabrics, wherein each of the k interconnection fabrics interconnectsless than all of the n nodes, and wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


24. The interconnection system of claim 23, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


25. The interconnection system of claim 23, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


26. The interconnection system of claim 23, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


27. The interconnection system of claim 23, wherein the kinterconnection fabrics provide redundant interconnection paths betweenevery pair of nodes from among the n nodes.
 28. A multi-fabricinterconnection system, comprising: k interconnection fabrics, eachhaving a vertex cardinality, m; and n nodes each having p ports, whereinp is greater than two, and wherein every pair of nodes from among the nnodes is interconnected through at least one of the k interconnectionfabrics, wherein every pair from among the p ports is interconnected toat least one of k interconnection fabrics, wherein each of the kinterconnection fabrics interconnects less than all of the n nodes, andwherein the vertex cardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


29. The interconnection system of claim 28, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


30. The interconnection system of claim 28, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


31. The interconnection system of claim 28, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


31. The interconnection system of claim 28, wherein the kinterconnection fabrics provide redundant interconnection paths betweenevery pair of nodes from among the n nodes.
 32. A multi-fabricinterconnection system, comprising: k incomplete interconnection fabricseach having a vertex cardinality, m; and n nodes each having p ports,wherein p is greater than two, and wherein every pair of nodes fromamong the n nodes is interconnected through at least one of the kinterconnection fabrics.
 33. The interconnection system of claim 32,wherein the vertex cardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


34. The interconnection system of claim 32, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


35. The interconnection system of claim 32, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


36. The interconnection system of claim 32, wherein the number k of theinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


37. The interconnection system of claim 32, wherein the kinterconnection fabrics provide redundant interconnection paths betweenevery pair of nodes from among the n nodes.
 38. A method for designing amulti-fabric interconnection system, comprising: interconnecting kfabrics each having a vertex cardinality, m; and interconnecting n nodeseach having p ports, wherein p is greater than two, and wherein everypair of nodes from among the n nodes is interconnected through at leastone of the k interconnection fabrics, wherein each of the kinterconnection fabrics interconnects less than all of the n nodes. 39.The method of claim 38, wherein the number k of interconnection fabricsis bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


40. The method of claim 38, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


41. The method of claim 38, wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


42. The method of claim 38, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


43. The method of claim 38, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 44. A method for designing a multi-fabric interconnection system,comprising: interconnecting k fabrics each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics,wherein each of the k interconnection fabrics interconnects less thanall of the n nodes, and wherein the number k of interconnection fabricsis bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


45. The method of claim 44, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


46. The method of claim 44, wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


47. The method of claim 44, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


48. The method of claim 44, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 49. A method for designing a multi-fabric interconnection system,comprising: interconnecting k fabrics, each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics,wherein every pair from among the p ports is interconnected to at leastone of k interconnection fabrics, wherein each of the k interconnectionfabrics interconnects less than all of the n nodes, and wherein thenumber k of interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


50. The method of claim 49, wherein the number k of the interconnectionfabrics is bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


51. The method of claim 49, wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


52. The method of claim 49, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {2{\left\lceil \frac{n + p - 1}{p} \right\rceil.}}$


53. The method of claim 49, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 54. A method for designing a multi-fabric interconnection system,comprising: interconnecting k fabrics each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics.55. The method of claim 54, wherein the number k of interconnectionfabrics is bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


56. The method of claim 54, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


57. The method of claim 54, wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


58. The method of claim 54, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


59. The method of claim 54, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 60. A method for designing a multi-fabric interconnection system,comprising: interconnecting k fabrics each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics,wherein each of the k interconnection fabrics interconnects less thanall of the n nodes, and wherein the vertex cardinality, m, is boundedabove by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


61. The method of claim 60, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


62. The method of claim 60, wherein the number k of interconnectionfabrics is bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


63. The method of claim 60, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


64. The method of claim 60, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 61. A method for designing a multi-fabric interconnection system,comprising: interconnecting k fabrics, each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics,wherein every pair from among the p ports is interconnected to at leastone of k interconnection fabrics, wherein each of the k interconnectionfabrics interconnects less than all of the n nodes, and wherein thevertex cardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


62. The method of claim 61, wherein the vertex cardinality, m, isbounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


63. The method of claim 61, wherein the number k of interconnectionfabrics is bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


64. The method of claim 61, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


65. The method of claim 61, wherein the k fabrics provide redundantinterconnection paths between every pair of nodes from among the nnodes.
 66. A method for designing a multi-fabric interconnection system,comprising: interconnecting k incomplete fabrics each having a vertexcardinality, m; and interconnecting n nodes each having p ports, whereinp is greater than two, and wherein every pair of nodes from among the nnodes is interconnected through at least one of the k interconnectionfabrics.
 67. The method of claim 66, wherein the vertex cardinality, m,is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


68. The method of claim 67, wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


69. The method of claim 67, wherein the number k of interconnectionfabrics is bounded above by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


70. The method of claim 67, wherein the number k of the interconnectionfabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


71. The method of claim 67, wherein the k incomplete fabrics provideredundant interconnection paths between every pair of nodes from amongthe n nodes.
 72. A computer-readable medium embodying program code fordesigning a multi-fabric interconnection system having instructions thatcause a computer to perform steps, comprising: interconnecting k fabricseach having a vertex cardinality, m; and interconnecting n nodes eachhaving p ports, wherein p is greater than two, and wherein every pair ofnodes from among the n nodes is interconnected through at least one ofthe k interconnection fabrics, wherein each of the k interconnectionfabrics interconnects less than all of the n nodes.
 73. Thecomputer-readable medium of claim 72, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


74. The computer-readable medium of claim 72, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


75. The computer-readable medium of claim 72, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


76. The computer-readable medium of claim 72, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


76. The computer-readable medium of claim 72, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 77. A computer-readable medium embodying program codefor designing a multi-fabric interconnection system having instructionsthat cause a computer to perform steps, comprising: interconnecting kfabrics each having a vertex cardinality, m; and interconnecting n nodeseach having p ports, wherein p is greater than two, and wherein everypair of nodes from among the n nodes is interconnected through at leastone of the k interconnection fabrics, wherein each of the kinterconnection fabrics interconnects less than all of the n nodes, andwherein the number k of interconnection fabrics is bounded above by therelationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


78. The computer-readable medium of claim 77, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


79. The computer-readable medium of claim 77, wherein the vertexcardinality, m, is bounded above by the relationship$m \geq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


80. The computer-readable medium of claim 77, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


81. The computer-readable medium of claim 77, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 82. A computer-readable medium embodying program codefor designing a multi-fabric interconnection system having instructionsthat cause a computer to perform steps, comprising: interconnecting kfabrics, each having a vertex cardinality, m; and interconnecting nnodes each having p ports, wherein p is greater than two, and whereinevery pair of nodes from among the n nodes is interconnected through atleast one of the k interconnection fabrics, wherein every pair fromamong the p ports is interconnected to at least one of k interconnectionfabrics, wherein each of the k interconnection fabrics interconnectsless than all of the n nodes, and wherein the number k ofinterconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


83. The computer-readable medium of claim 82, wherein the number k ofthe interconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


84. The computer-readable medium of claim 82, wherein the vertexcardinality, m, is bounded above by the relationship$m \geq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


85. The computer-readable medium of claim 82, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


86. The computer-readable medium of claim 82, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 87. A computer-readable medium embodying program codefor designing a multi-fabric interconnection system having instructionsthat cause a computer to perform steps, comprising: interconnecting kfabrics each having a vertex cardinality, m; and interconnecting n nodeseach having p ports, wherein p is greater than two, and wherein everypair of nodes from among the n nodes is interconnected through at leastone of the k interconnection fabrics.
 88. The computer-readable mediumof claim 87, wherein the number k of interconnection fabrics is boundedabove by the relationship $k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


89. The computer-readable medium of claim 87, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


90. The computer-readable medium of claim 87, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


91. The computer-readable medium of claim 87, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


92. The computer-readable medium of claim 87, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 93. A computer-readable medium embodying program codefor designing a multi-fabric interconnection system having instructionsthat cause a computer to perform steps, comprising: interconnecting kfabrics each having a vertex cardinality, m; and interconnecting n nodeseach having p ports, wherein p is greater than two, and wherein everypair of nodes from among the n nodes is interconnected through at leastone of the k interconnection fabrics, wherein each of the kinterconnection fabrics interconnects less than all of the n nodes, andwherein the vertex cardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


94. The computer-readable medium of claim 93, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


95. The computer-readable medium of claim 93, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


96. The computer-readable medium of claim 93, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


97. The computer-readable medium of claim 93, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 98. A computer-readable medium embodying program codefor designing a multi-fabric interconnection system having instructionsthat cause a computer to perform steps, comprising: interconnecting kfabrics, each having a vertex cardinality, m; and interconnecting nnodes each having p ports, wherein p is greater than two, and whereinevery pair of nodes from among the n nodes is interconnected through atleast one of the k interconnection fabrics, wherein every pair fromamong the p ports is interconnected to at least one of k interconnectionfabrics, wherein each of the k interconnection fabrics interconnectsless than all of the n nodes, and wherein the vertex cardinality, m, isbounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


99. The computer-readable medium of claim 98, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


100. The computer-readable medium of claim 98, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


101. The computer-readable medium of claim 98, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


102. The computer-readable medium of claim 98, wherein the k fabricsprovide redundant interconnection paths between every pair of nodes fromamong the n nodes.
 103. A computer-readable medium embodying programcode for designing a multi-fabric interconnection system havinginstructions that cause a computer to perform steps, comprising:interconnecting k incomplete fabrics each having a vertex cardinality,m; and interconnecting n nodes each having p ports, wherein p is greaterthan two, and wherein every pair of nodes from among the n nodes isinterconnected through at least one of the k interconnection fabrics.104. The computer-readable medium of claim 102, wherein the vertexcardinality, m, is bounded above by the relationship$m \leq {2{\left\lceil \frac{n}{p + 1} \right\rceil.}}$


105. The computer-readable medium of claim 102, wherein the vertexcardinality, m, is bounded below by the relationship$m \geq {\left\lceil \frac{n + p - 1}{p} \right\rceil.}$


106. The computer-readable medium of claim 102, wherein the number k ofinterconnection fabrics is bounded above by the relationship$k \leq {\begin{pmatrix}{p + 1} \\2\end{pmatrix}.}$


107. The computer-readable medium of claim 102, wherein the number k ofthe interconnection fabrics is bounded below by the relationship$k \geq {\left\lceil \frac{\begin{pmatrix}n \\2\end{pmatrix}}{\begin{pmatrix}m \\2\end{pmatrix}} \right\rceil.}$


108. The computer-readable medium of claim 102, wherein the k incompletefabrics provide redundant interconnection paths between every pair ofnodes from among the n nodes.
 109. A multi-fabric interconnectionsystem, comprising: ten interconnection fabrics; and five nodes eachhaving p ports, wherein p is greater than two, and wherein every pair ofnodes from among the five nodes is interconnected through at least oneof the ten interconnection fabrics, wherein each of the teninterconnection fabrics interconnects less than all of the five nodes.110. The interconnection system of claim 108, wherein the teninterconnection fabrics provide for fault-tolerance.
 111. Theinterconnection system of claim 108, wherein the ten interconnectionfabrics provide for redundant interconnection paths.